effici learn accuraci label sourc select sampl pinar donmez jaim carbonel jeff schneider scalabl data mine task reli activ learn provid accur label instanc multipl label sourc oracl expert unknown reliabl recent advent inexpens scalabl onlin annot tool amazon mechan turk label process vulner nois prior knowledg accuraci individu label paper address challeng jointli learn accuraci label sourc obtain inform label activ learn task hand minim total label effort specif present iethresh interv estim threshold strategi intellig select expert highest estim label accuraci iethresh estim confid interv reliabl expert filter estim upper bound confid interv threshold jointli optim expect accuraci estim expert accuraci varianc framework flexibl work wide rang nois level outperform baselin expert random expert select iethresh achiev level accuraci half queri issu expert label queri requir random expert select dataset uci mushroom result show method natur balanc explor exploit gain knowledg expert reli select increas frequenc 